Enhanced Error Correction via Language Processing

There are two fundamental approaches for error correction. One approach is to add external redundancy to data. The other approach is to use the redundancy inside data, even if it is only the residual redundancy after a data compression algorithm. The first approach, namely error-correcting codes (ECCs), has been studied actively over the past seventy years. In this work, we explore the second approach, and show that it can substantially enhance the error-correction performance. There are good practical reasons to study the second approach. In data storage, including flash-memory systems, the amount of data institutions and individuals store keeps growing fast, and our dependency on stored data has never been higher. It is necessary to keep data highly reliable (ideally, never to be lost), so all means are needed for this goal. The two approaches – ECC and using internal redundancy – are compatible because the latter requires no change to the stored data. At normal times, the storage system can correct errors as usual using ECCs. But if errors exceed the correction capability of the ECC, – which is a rare but important event, – the second approach can be combined with ECC for a more powerful error correction performance. This way, a better balance between data reliability and average efficiency can be achieved. This work focuses on error correction of texts in English as a case study. (For an illustrative example, see the full paper of this work [2].) It proposes a scheme that combines languagebased decoding with ECC decoding. Both analysis and experimental results are presented, and in particular, we have tested its experimental performance in a flash memory testbed under highly practical settings. The tested ECCs are LDPC codes, a family of capacity-approaching codes important for memories, and our results show that language-based decoding can very significantly enhance the error correction performance of LDPC codes by providing useful soft information. This work is a continuation of Shannon’s well known 1951 paper on the entropy of the English language. It is also related to our prior work [3], where a decoding algorithm by dynamic programming was used. The work here is new in several significant ways. First, its decoding is based on new algorithms and techniques, which achieves faster decoding speed and more robust decoding performance. Second, it uses soft decoding instead of hard decoding, with a clear